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ABSTRACT 

Under the sponsorship of the National Science Foundation, 22 
urban school districts have been involved in a long-term educational reform 
through the Urban Systemic Initiative (USI) program since 1994. This paper 
presents a brief summary of findings regarding best policies and practices in 
the educational reform effort focusing on student achievement in mathematics 
and science. The USI has been based on six educational reform drivers: four 
process drivers and two student outcome drivers. The evaluative study team 
collected quantitative and qualitative data from 22 USI sites using the Key 
Indicator Data Collection System, a district-level data collection instrument 
based on a cross-site longitudinal evaluative framework. The data collection 
effort was supplemented by interviews, document reviews, and site visits. The 
evaluation confirmed positive linkages between the USI policy implementation 
rubrics and student outcomes. The six reform drivers provided a visionary 
direction for systemwide educational reform for most USI sites. The four 
process drivers worked dynamically together to improve the two outcome 
drivers— student achievement and gap elimination— that are the overarching 
goals of the systemic initiatives, A graphical representation shows the links 
among rubrics and outcomes as a "schoolhouse" in which the foundation is the 
Belief System. (SLD) 
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BEST POLICIES AND PRACTICES IN URBAN EDUCATIONAL REFORM: 
A SUMMARY OF EMPIRICAL ANALYSIS 
FOCUSING ON STUDENT ACHIEVEMENTS AND EQUITY 



Jason J. Kim, Ph.D. 
Linda M. Crasco 
Systemic Research, Inc. 



BACKGROUND AND OBJECTIVE 

Under the sponsorship of the National Science Foundation (NSF), 22 urban school districts have 
been involved in long term educational reform through the Urban Systemic Initiative (USI) 
program since 1994. USI has been a reform catalyst for large-scale educational change affecting 
standards, curriculum, assessment, professional development, partnerships, and convergence of 
intellectual and fiscal resources, with constant attention to improving student achievement in 
mathematics and science. This paper presents a brief summary of findings regarding best 
policies and practices in the educational reform effort focusing on student achievement in 
mathematics and science. 

A theoretical systemic educational reform model has evolved based on the six “educational 
reform drivers” including four process drivers (standards-based curriculum, instruction and 
assessment; policy; resources; and broad-based support) and two student outcome drivers 
(student achievement; and improvement of historically underserved). This evaluative study 
explored causal inferential models among the process drivers (independent variables) and 
outcome drivers (dependent variables) linking policy implementation rubrics to quantitative 
student outcome data. 



METHODOLOGY AND DATA SOURCES 

The evaluative study team has designed and collected quantitative and qualitative data from 22 
USI sites* using the Key Indicator Data Collection System (KIDS), a district level data 
collection instrument based on a cross-site, longitudinal evaluative study framework. The data 
collection effort has been supplemented by school district document reviews, and site visits that 
included interviews, focus groups, classroom and professional development surveys and teacher 
surveys. The quantitative and qualitative data have been compiled and analyzed using statistical 
regression methods to link policies and practices to student outcomes. 

The KIDS Part 1 Quantitative (K-1) templates collected data from the individual project from the 
baseline year to school year 1999-2000. K-1 includes: district-wide student demographics and 
selected statistics; mathematics and science gate-keeping course enrollment and completion; 12th 
grade students graduation data summary; high school graduation requirements; AP mathematics 
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and science test results; SAT; ACT; assessment test results; teacher certificates and professional 
development (Ref 1 & 4). 

The KIDS Part 2 Qualitative (K-2) templates collected data relevant to policy implementation 
over the project period, including: policies promoting equal access by all students in high quality 
education; policies impacting the enrollment of students in challenging math and science 
courses; policies relevant to standards-based curriculum and instruction; policies relevant to 
teacher qualifications; professional development policies and practices; policies relevant to 
standards-based assessments; USI leadership, governance, and management; and partnerships, 
changes in policies and practices were tracked over the years of program implementation. 

The K-2 qualitative data was further compiled as 46 rubric elements that reflect the four process 
drivers of systemic reform. For the development of causal inferential model, the 46 rubric 
elements were combined to form 14 policy rubric groups as shown in Table 1. Statistical 
methods, including factor analysis and regression, were used to explore the linkages between 
polices and outcomes (independent variables) and student achievement outcomes (dependent 
variables.) 



Table 1 

Education Reform Qualitative Rubric Groups and Elements 



1. Equal Access 

1. 1 Ehmtnation ol Tracking 

1.2 More Upper Love! Courses 

2. Targeted Programs 

2.1 Slratogios to Addross Noeds ol Special Populations 

2 2 Targeted Programs for Undorreprosontocl Minontios 

3. Graduation and Promotion Policies 

3.1 Graduation Requirements Moio Rigorous than the State 

3.2 Graduation Requirements of at Least 3 Years Mathematics and 
Science 

3 3 Promotion Policy 

4. Altendanco and Safety Net Programs 

4 1 Auoiidanco Policy 
4 2 Summer Scitool 

4.3 Tutoring or Saturday Academy 

5. StandardS'Basad Curriculum 

5.1 Standards-Ba.sod Curriculum 

5.2 Curriculum Linked to State or National Content Standards 

5.3 Significant and Increasing Number ol Sdiools tmplomcnting 
Standards Based Curriculum or Matciials 

6. Assessment Alignment and InstructionalTime 

6.1 Asscssntonl Adequately Aligned to Curriculum and Standards 

6.2 More Instructional Timo in Mathomatres and Science 

7. Toachar Certification 

7.1 Policios Guiding Toaclioi Assignmciits 

72 Uncertified Toachcrs Encouraged to Souk Ccarfication 

73 Teacfior Certification Required by tho District 

8. Professional Development Content and Stondards 

8.1 Reward for Increasing Content Area Knowledge 

8.2 Professional Development Content 

8 3 Aligning Professional Development willi Standards 

8.4 Professional Development Offerings 



9. Professional Development Participation and Follow-up 

9.1 Professional Development Participation 

9.2 Classroom Observation 

9 3 Evaluation by Analysis ot Student Scores 

9.4 Professional Development Follow-Up 

10. Leadership, Governance and Managament 

10.1 Supenntendont's Loadorship and Tenure 

10.2 USI Proiect Director's Position/Tenure in District Organizational 
Structure 

10.3 USI StafI 

11. Collaboration, Partnarships with Parants, Htghar Education, 
Business and Partners for Students 

11 1 Collaboration with Other Initiativoa 

11 2 Community or Parental Involvement 

11.3 Partnerships with Higher Education 

114 Partnerships with Industry and Business 
11.5 Partnerships Directly Linked to Students 

12. Assessment Policios and Uso of Rosults 

12.1 Required State Assessment 

12.2 Local Standards- Baaed Assessment Measures 

12 3 Otliur Standard Measures Used 

12.4 Using Assessment for Planning Instiuction 

12.5 Usage of Student Tost Rosults lor Curriculum and Instruction 
Enhancement 

13. Accountabiltty and Usa of District Data 

13. 1 Report Card System for Stakeholders 

13 2 Data Collodion Infrastructure 

13.3 Usago of Key Indicator Data lor USI Program Inter venttons 

14. Coordinotion, Loveraga of Resources, end Use of Data and 
Technology 

14.1 Coordination Among Existing Funding Sources 

14 2 Leverage of Funds 

14.3 Usool Researdi Rosults 

14 4 Todmology anclTelocor nmunications 
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SUMMARY OF FINDINGS 



Student Achievements and Equity 

NSF’s six drivers of systemic reform provided a framework for USI implementation, especially 
focusing on standards-based curriculum and instruction, policies supporting all students in high 
quality mathematics and science education, professional development, aligned assessment 
instruments, convergence of resources, leadership, and partnerships. These advances are 
accompanied by evidence that urban school students show gains in Mathematics and Science 
achievements. Among the findings from the trend analyses of 21 USI sites: 



1. As shown in Exhibit I, urban students in the USI school districts have dramatically 

increased their enrollment rates in mathematics and science gate-keeping and higher-level 
courses. Larger participant gains are observed in more mature USI cohorts. For 
example, enrollment in gate-keeping and higher-level mathematics courses has increased 
in four Cohort 93 sites (Chicago, Dallas, Miami-Dade, and New York), from 29% of the 
total high school student population in SY 1993-94 to 49% in SY 1999-00. 



Exhibit I 

Mathematics Enrollment and Completion in 
Grade 9*12 Gate-Keeping and Higher-Level Courses' 
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Science gate-keeping course enrollments in the same sites have experienced even higher 
gains over the same project period; from 59% to 80% of the total student population. 

2. Underrepresented minority students made greater enrollment gains than their peers during 
the same period, resulting in reduced enrollment disparities as shown in Exhibit II. For 
example, in four Cohort 93 sites (Chicago, Dallas, Miami-Dade, and New York), the 



Number of Enrolled per 1,000 



enrollment disparity between African-American and white students in gate-keeping and 
higher-level science courses narrowed from 179 per 1,000 to 146 per 1,000 between SY 
1993-94 and 1999-00. The Hispanic to white student disparity narrowed from 258 to 205 
students per 1 ,000. 



Exhibit II 

Reduced Disparity and Increased Enrollment in Grade 9-12 Science Gate-Keeping 
and Higher-Level Courses^: Number of Students Enrolled in Science per 1,000 Students 
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Hispanic-to-White Disparity (Cohort 93) 
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* Includes Biology I, Physics I, and Integrated Science l-lll. Data includes Chicago, Dallas, Mia mi-Dade, and New York. 



3. Assessment test results show that USI students have made gains in mathematics and 
science achievement, while reducing achievement gaps among racial and ethnic groups. 
For example, passing rates in 8th grade science assessment tests improved in 15 out of 
the 16 sites with data available. Seven of these sites exhibited a narrowing of the passing 
rate gap between underrepresented minority and white students. Eighth grade 
mathematics assessment test results also show impressive passing rate improvements; 16 
out of 17 sites experienced overall improvement with eight sites narrowing the 
achievement gap (Exhibit not shown here). 

4. The increasing numbers of 1 1th and 12th grade students taking college entrance 
examinations indicate more students have aspirations of pursuing post-secondary 
education. By SY 1999-00, Cohort 93 students were taking AP mathematics and science 
examinations at a rate higher than the national average (in case of mathematics, 35.9 per 
1,000 students compared to the national average of 22.9). Cohort 94 students also follow 
the same trend. SAT and ACT test-taking rates show a similar pattern. These trends are 
most notable in the cohorts with the longest participation in the USI program (Exhibit not 
shown here). 








Impact of Policy Implementation 

Student achievement gains were linked to policy implementation rubrics based on qualitative 
data from K-2, site visit, and other supplemental district level reports. As an example, Exhibit 
III shows the relationship between student enrollment in gate-keeping and higher-level 
mathematics courses. Algebra II, Geometry and Calculus (dependent variable) and the “open 
access” policy implementation rubric (independent variable). 

The X axis represents the annualized percentage point (pp) change in mathematics enrollment. 
The annualized percentage point change is the average yearly change in percentage of total high 
school students enrolled in higher mathematics from the baseline to SY 1999-00. For example, a 
site with a baseline year of 1993-94 and an enrollment increase during the six years from 30% to 
60% of the total high school student population will have an annualized pp increase of 5 pp. The 
Y axis represents the USI sites’ policy rubric scores weighted by the implementation time factor. 
The “open access” policy rubric is a composite index of two rubric elements: elimination of 
tracking, and more availability of higher level courses. 

The regression equation shows a positive relationship: Y = -0.006545 + 0.034996*X. The 
estimators are not statistically significant (F-test P value = 0.090) due to the limited sample size 
(10 sites data), however, these 10 data points represent mathematics enrollment gains among 
more than 3.2 million students. 

Exhibit III 

Higher Math Enrollment (Annualized Percentage Point Change) vs Equal Access Rubric 



Regression Equation: Y = -Q.QQB545 + □.□3400B*X 
F-test P Value = 0.000 
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Note: 

1 . Size of bubble represents relative size of total high school student enrollment. 

2. Higher Math: Algebra II.Geometry.Calculus.and Integrated Math ll-IV. 

3. The annualized percentage point (pp) change Indicates the average yearly pp change in percentage of total high school 
students enrolled in higher mathematics from the baseline to SY 1 999-00. For example, a site with a baseline year of 
1993-94 and an enrollment increase during the six years from 30% to 60% of the total high school student population 
will have an annualized pp Increase of 5 pp. 

4. The rubric scores have been weighted by the Implementation time factor (number of years policy in effect). 
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A regression analysis of student enrollment in gate-keeping and higher-level science courses and 
the “open access” policy implementation rubrics also indicates a positive relationship (not shown 
here). The regression equation is Y = 0.007413 + 0.02663*X; a positive coefficient. As with the 
example shown above, the estimators are not statistically significant (F-test P value = 0.2279) 
due to the limited sample size (12 sites data points), however, these 12 data points represent 
science enrollment gains among close to 3.4 million students. 

When we examined the relationships among the 14 rubric groups, we observed that many rubrics 
groups are strongly correlated in educational reform policy implementation. For example. 
Exhibit rv presents the relationships among three rubric groups: Equal Access (Rubric Group 1 
in Table 1), Targeted Programs (Rubric Group 2), and Graduation and Promotion Policies 
(Rubric Group 3) using a three dimensional bubble chart. Tliree axis present scores of the three 
respective rubrics. Each bubble represents a USI site, and the size of bubble indicates the 
relative size of the total district student enrollment. The majority of USI sites strongly support 
“equal access” and “graduation and promotion policy”, while showing a wide variation in 
“targeted programs.” 



Exhibit IV 

Relationship ofThree Rubrics: Equal Access, Targeted Programs, 
and Graduation and Promotion Policies 
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CONCLUSION 

The preliminary study report published in June 2001 “Academic Excellence for All Urban 
Students- Their Accomplishment in Science and Mathematics” confirmed evidence of 
noteworthy gains in student achievement (Ref 2). This report confirms positive linkage between 
policy implementation rubrics and student outcomes. 

NSF’s six drivers provided a visionary direction for system wide educational reform for most 
USI sites. The four process drivers dynamically work together to improve the two outcome 
drivers- student achievement and gap elimination- which are the overarching goals of the 
systemic initiatives. Based on findings from the causal inferential models discussed above. 
Exhibit V presents a graphical representation of “How Reform Works,” linking rubrics and 
outcomes. In the schoolhouse diagram, the building blocks are the process and outcome drivers. 
While each block is an important individual component of the system, all must interact with each 
other in a balanced manner to support the goals of the system. 

The foundation of the reform effort is the “Belief System”: the expectation that all students can 
and must learn challenging mathematics and science content, and must be held to the same high 
standards. Another layer of the foundation is Policies to Support Science and Mathematics 
Achievement and Equity (Driver 2). These policies are key to ensuring that teachers, 
administrators and staff throughout the system provide the same services and resources to all 
students. 

The pillars of the schoolhouse are four supporting building blocks, on top of the “Belief System 
and Supporting Policy” foundation: 

• Standards-Based Instruction, Curriculum, Curriculum Framework, and Science and 
Mathematics Standards (Driver 1) 

• Standards-Based Assessment and Accountability (Driver 1) 

• Convergence of Resources: Material, Fiscal and Intellectual (Driver 3) 

• Broad-based Support: Leadership, Governance and Management; Partnerships with 
Higher Education, Business, Parents and Community (Driver 4) 

Overlying the supporting structure is the Professional Development and Teacher Certification/ 
Qualification (Driver 2) building blocks. The delivery of the curriculum is accomplished 
through the efforts of teachers in the system. Students need to be taught by well-qualified 
teachers. Continued professional development is necessary as new standards and curricula are 
introduced, and as research demonstrates which teaching practices allow students to reach their 
full potential (Ref 3 & 5). 

The next level is the Enacted Curriculum in Classrooms and Student Support Programs (Driver 
2). Enacted curriculum is the actual teaching of the prescribed curricula in the classroom. It 
consists of the content material and the methods used to present the subject matter to students. 
Student support systems become more important as all students are exposed to gate-keeping and 
higher-level courses with challenging content areas. Examples of student support systems are 
tutoring and summer enrichment programs. 
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Exhibit V 

SYSTEMIC BUILDING BLOCKS FOR URBAN SCHOOL REFORM: 
LINKAGE OF DRIVERS, RUBRICS, AND OUTCOMES 
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Belief System - High Expectations For All Students 



The bell tower of our schoolhouse symbolizes the ultimate goal of school reform - Student 
Outcomes: Science and Mathematics Achievement and Participation and Achievement Gaps 
Narrowed (Drivers 5 and 6). The goal of the systemic educational reform is to prepare students 
for higher education and career opportunities in the area of Science, Technology, Engineering, 
and Mathematics (STEM). Successfully attaining this goal benefits individual students and 
society as a whole. 



** The authors acknowledge the support of the National Science Foundation through Grant No. REC-9874322, 
“How Reform Works: An Evaluative Study ofNSF’s Urban Systemic Initiative.” Any opinions, findings and 
conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the 
views of the National Science Foundation. 

Cohort 93 (Baltimore, Chicago, Dallas, Detroit, El Paso, Miami-Dade, New York, Phoenix); Cohort 94 
(Cleveland, Columbus, Fresno, Los Angeles, Memphis, New Orleans, Philadelphia); Cohort 95 (Milwaukee, St. 
Louis, San Antonio, San Diego); Cohort 97 (Atlanta, Jacksonville); Cohort 99 (Houston) 
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